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Usability  and  Effectiveness  oe  Advanced  General  Aviation  Cockpit 
Displays  eor  Instrument  Flight  Procedures 


introduction 


More  than  three  years  ago,  the  Federal  Aviation 
Administration  instituted  the  Alaska  Capstone  Pro¬ 
gram  in  an  effort  to  improve  flight  safety  in  Alaska 
(FAA,  2003).  The  Capstone  Program  provided  for 
the  installation  of  new  cockpit  avionics  for  over  150 
general  aviation  (GA)  aircraft  in  the  southwest  region 
of  Alaska,  centered  around  the  town  of  Bethel.  The 


avionics  system,  developed  by  UPS  Aviation  Technolo¬ 
gies,  consisted  of  a  multi -function  display  (MFD)  unit, 
the  Apollo  MX-20,  and  an  accompanying  Global  Po¬ 
sitioning  System  (GPS)  display,  the  Apollo  GX-60.  In 
addition,  each  aircraft  was  equipped  with  a  Universal 
Access  Transceiver  (UAT) ,  which  provided  datalink  com¬ 
munication  between  the  aircraft  and  a  ground  station 


Figure  1 :  Capstone  Avionics  displays.  The  MX-20  display  is  shown  on  top  and  the  GX-60 
GPS  unit  is  shown  below  it. 
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or  from  one  aircraft  to  another  aircraft.  Figure  1  shows 
the  Capstone  avionics  displays. 

The  two  Capstone  displays  provided  pilots  with  a  mov¬ 
ing  map  that  shows  ownship  (i.e.,  own  aircraft)  position. 
Routes  entered  into  the  GPS  display  are  presented  as  a 
magenta  line  on  the  moving  map.  The  map  can  display 
an  instrument  flight  chart,  with  airways,  intersections, 
airports,  and  other  navigational  points  or  a  visual  flight 
rules  (VFR)  sectional  chart  that  includes  terrain  features. 
Many  types  of  information  can  be  overlaid  on  the  map 
at  the  pilot’s  discretion.  For  example,  the  display  can  also 
show  the  relative  height  of  terrain  in  the  form  of  red, 
green,  yellow,  or  black  colored  blocks.  Using  a  custom 
map  page,  pilots  can  overlay  terrain  with  airport  and 
other  information.  The  display  also  provides  traffic  and 
weather  information  to  the  pilot.  Traffic  information  is 
dependent  on  a  ground/ air/space  infrastructure  known 
as  Automatic  Dependent  Surveillance  -  Broadcast,  or 
ADS-B.  In  addition,  weather  information  is  uplinked 
to  the  pilot  through  a  system  known  as  Flight  Infor¬ 
mation  Services  -  Broadcast,  or  FIS-B,  from  weather 
reporting  stations. 

Interviews  with  pilots  participating  in  the  Capstone 
Program  have  provided  significant  insight  into  the  impact 
of  this  technology  on  flight  operations  (Williams,  Yost, 
Ffolland,  &  Tyler,  2002).  The  pilots  interviewed  have 
been  mostly  positive.  Still,  some  system  features  have  been 
identified  as  needing  improvement.  While  information 
gained  from  the  interviews  will  assist  in  the  transition  of 
this  type  of  technology  to  aircraft  in  the  rest  of  the  U.S., 
individuals  participating  in  the  Capstone  Program  were 
all  14  CFRPart  135  pilots  (i.e.,  professional  pilots),  flying 
almost  every  day  of  the  year.  Consequently,  the  majority 
of  these  pilots  fly  over  100  hours  each  month,  and  nearly 
all  (95%)  are  instrument  rated. 

This  leads  to  the  obvious  question  of  the  impact  of 
this  kind  of  technology  on  the  14  CFRPart  91  (i.e.,  non¬ 
professional)  pilot  population.  Part  9 1  pilots  typically  fly 
much  less  often,  and  a  significant  portion  of  this  popula¬ 
tion  does  not  have  an  instrument  rating  (FAA,  2000).  In 
addition,  the  Part  9 1  population  is  more  diverse,  having 
a  wider  range  of  total  flight  experience  and  ages.  What  is 
needed  is  an  examination  of  the  effect  of  these  displays 
on  the  Part  91  pilot  population.  The  present  study  was 
designed  to  identify  human  factors  that  should  be  con¬ 
sidered  during  the  deployment  of  this  technology  to  the 
entire  general  aviation  community  and  in  the  develop¬ 
ment  of  future  displays. 

When  examining  Part  91  flight  activities,  two  pilot 
groups  should  be  considered  before  transitioning  this 
equipment  to  the  GApilot  population.  One  group  consists 
of  low-time  pilots  who  would  only  use  the  displays  in 
VFR  conditions  because  they  do  not  have  an  instrument 


rating.  While  their  flight  tasks  are  simpler,  they  typically 
have  fewer  resources  to  assist  them  and,  as  such,  repre¬ 
sent  an  interesting  group  for  study.  A  future  study  will 
examine  this  pilot  group. 

The  other  group  consists  of  pilots  who  will  use  the 
multi-function  navigational  displays  in  instrument  flight. 
Instrument  flight  is  one  of  the  most  complex  aspects  of 
single-pilot  operations.  This  complexity  is  exacerbated 
when  using  a  GPS  display  because  of  the  type  and  amount 
of  user  interaction  required  to  operate  the  equipment. 
Prior  research  has  indicated  that  GPS  displays  are  among 
the  most  complex  equipment  that  most  Part  91  pilots 
interact  with  in  the  cockpit  (Williams,  1999a),  and  that 
GPS  user-interface  design  deficiencies  tend  to  exacerbate 
the  difficulty  of  using  these  displays  (Nendick,  1995; 
Nendick  &  St. George,  1995;  Heron  &  Nendick,  1999; 
Wickens,  2000;  Williams,  1999a,  1999b).  The  current 
study  was  designed  to  examine  the  usability  of  these 
advanced  displays  for  instrument  flight  activities  and  to 
compare  the  effect  that  the  displays  have  during  instru¬ 
ment  flight  on  pilot  performance  and  workload  relative 
to  standard  navigation  instruments. Thestudyfocuses  on 
single-pilot  operations  during  normal  to  high  workload 
conditions,  including  a  failure  of  the  vacuum-driven 
cockpit  displays. 

METHODS 

Participants 

Sixteen  instrument- rated  pilots  were  recruited  from  the 
Oklahoma  City  metropolitan  area  and  were  compensated 
monetarily  for  their  participation.  All  of  the  pilots  were 
male,  with  the  average  age  of  29.75  years,  a  median  age 
of  25  years,  and  a  range  in  ages  from  21  to  69  years.  The 
participants  were  asked  to  estimate  the  number  of  flight 
hours  for  several  categories  of  flight.  They  were  also  asked 
whether  they  had  used  a  cockpit  GPS  or  multi-function 
display  and  if  they  had  used  a  flight  simulation  program 
similar  to  the  one  employed  in  the  study.  This  informa¬ 
tion  is  summarized  in  the  Results  section. 

Design 

In  the  current  study,  pilots  were  asked  to  plan  and  fly 
two  separate  flights  in  instrument  conditions.  During 
one  flight,  pilots  were  permitted  to  use  only  a  standard 
navigational  instrument  (i.e.,  VOR),  instrument  charts, 
and  sectional  charts.  During  the  other  flight,  pilots  were 
asked  to  conduct  the  flight  using  the  GX-50'  GPS  and 
MX-20  multi-function  displays.  Both  flights  included  a 


*  There  is  no  difference  in  the  user-interface  of  the  Apollo  GX-60 
and  GX-50  displays.  The  GX-60  display  has  additional  functionality 
that  was  not  required  for  the  experiment. 
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number  of  instrument  procedures  commonly  required 
during  instrument  flight,  as  well  as  a  vacuum  failure, 
designed  to  subject  the  pilot  to  a  high  workload  while 
interacting  with  the  displays. 

Measurement  Instruments 

Usability,  workload,  and  pilot  performance  were  gauged 
using  a  number  of  subjective  and  objective  measures. 
Usability  measures  primarily  focused  on  the  GX-50 
and  MX-20  displays.  No  evaluations  of  usability  of  the 
VOR  navigational  display  were  performed.  A  usability 
questionnaire  was  administered  to  each  pilot  to  gauge 
various  aspects  of  the  GX-50  and  MX-20  displays.  Six 
rating  criteria  (effectiveness,  ease  of  use,  workload,  cues 
and  prompts,  system  feedback,  and  head-down  time) 
were  scored  using  a  5-point  Likert  scale  for  each  of  the 
display’s  features  and  functions.  A  copy  of  the  question¬ 
naire  is  reproduced  in  the  appendix.  In  addition,  button 
presses  were  videotaped  to  measure  interaction  errors.  An 
experimenter  was  available  to  assist  the  pilot  on  use  of 
the  displays  during  the  flights.  Experimenter  assistance 
on  tasks  was  recorded. 

The  NASA  TLX  subjective  workload  questionnaire 
(Hart  &  Staveland,  1988)  was  administered  to  each  pilot 


to  get  a  subjective  measure  of  workload  for  several  of  the 
in-flight  tasks.  NASA  TLX  is  a  six-item  questionnaire 
measuring  mental  effort,  physical  effort,  temporal  pres¬ 
sure,  perceived  performance,  total  mental  and  physical 
effort,  and  frustration.  Immediately  after  a  selected  in¬ 
flight  task,  the  pilot  was  asked  to  give  a  score  (1  meaning 
low  and  10  meaning  high)  for  each  of  the  six  items. 

Equipment 

All  of  the  flights  were  conducted  in  the  Givil  Aero¬ 
space  Medical  Institute’s  Basic  General  Aviation  Research 
Simulator  (BGARS),  comprised  of  seven  networked 
PGs  using  Microsoft’s  Flight  Simulator  2002,  flve 
out-the-window  views,  and  moderate-fldelity  controls 
(Precision  Flight  Systems  —  Dual  Professional  Flight 
Gonsole).  Flight  performance  data  were  recorded  four 
times  per  second  and  consisted  of  latitude,  longitude, 
altitude,  ground  speed,  heading,  bank,  cross-track  error, 
elapsed  time,  and  flight  segment.  The  simulator  was 
conflgured  as  a  Gessna  1 82RG  for  this  study.  Figure  2 
shows  a  picture  of  the  simulator.  Figure  3  shows  how 
the  GPS  and  MFD  were  situated  in  relation  to  the  rest 
of  the  aircraft  instruments. 


Figure  2:  Basic  General  Aviation  Research  Simulator  (BGARS). 
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Figure  3:  Close-up  view  of  aircraft  instruments  and  GPS/MFD  displays  (to  the  right). 


Procedure 

Pilots  filled  out  the  required  consent  forms  and  then 
were  briefed  about  the  experimental  protocol.  Each  pilot 
received  approximately  two  to  three  hours  of  training  on 
the  GX-50  and  the  MX-20.  The  training  began  on  the 
GX-50  by  demonstrating  the  differences  between  the  two 
types  of  keys  (smart  and  hard  keys)  and  the  functions  as¬ 
sociated  with  each  key.  Specifically,  the  pilots  were  taught 
how  to  perform  a  series  of  tasks  that  they  later  would  be 
required  to  perform  during  the  flight.  During  the  train¬ 
ing,  the  pilots  were  required  to  demonstrate  minimal 
proficiency  with  the  GX-50,  measured  as  one  unassisted 
completion  of  the  task,  to  perform  the  following  tasks: 

•  Go  direct  to  a  waypoint; 

•  Extract  information  about  various  waypoints; 

•  Build,  edit,  and  modify  an  active  or  saved  flight 
plan; 

•  Navigate  using  the  course  deviation  indicator  (GDI) 
on  the  GPS  display; 

•  Set  up  and  enter  holding  procedures; 

•  Load  and  change  approaches;  and 

•  Perform  missed  approach  procedures. 

Next,  the  pilots  were  shown  how  to  use  the  MX-20.  The 
four  different  map  modes  (VFR,  IFR, Terrain,  and  Gustom) 
were  shown  to  the  pilots.  Within  each  mode,  the  menu 
structure  was  explained,  and  detailed  instructions  were 


given  on  how  to  select  the  map  orientation  (North-up, 
track-up,  and  desired  track-up)  and  appearance  (declut¬ 
ter  options  and  overlays).  Last,  the  flight  plan  page  was 
explained  to  each  of  the  pilots.  Breaks  were  provided  to 
the  participants  as  required. 

Following  the  training,  a  20-minute  warm-up  flight 
was  given  to  the  pilots  to  allow  time  for  familiarization 
with  the  simulator  and  to  be  sure  that  the  pilots  could 
perform  the  necessary  instrument  flight  tasks.  At  the 
conclusion  of  the  warm-up  flight,  a  15-minute  break 
was  given  to  each  pilot. 

The  experimental  procedure  required  that  the  pilots  fly 
two  similar  IFR  flight  plans  in  instrument  meteorological 
conditions.  When  following  one  flight  plan,  pilots  were 
required  to  navigate  using  only  the  GX-50  and  MX-20, 
while  the  other  flight  plan  required  the  pilots  to  navigate 
using  only  a  VOR  head.  GPS  approaches  were  flown 
with  the  GX-50  and  MX-20,  and  VOR  approaches 
were  flown  in  the  VOR  condition.  Pilots  had  access  to 
all  appropriate  paper  charts  (U.S.  Terminal  Procedures 
charts.  Sectional  Aeronautical  chart,  and  IFR  En  Route 
Low  Altitude  charts)  during  both  flights.  Pilots  were  free, 
both  before  and  during  the  flight,  to  set  up  and  adjust 
their  MX-20  map  display  at  their  discretion. 

The  first  flight  scenario  consisted  of  flying  from 
Elanscom  Field,  in  Bedford,  Massachusetts  (BED),  to 
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the  Gardner  (GDM)  VORTAG.  Shortly  after  takeoff, 
pilots  were  redirected  to  fly  the  100-degree  radial  in¬ 
bound  at  GDM  to  avoid  a  restricted  flight  zone.  Before 
reaching  GDM  pilots  were  given  instructions  to  hold 
at  the  HURLY  intersection  on  the  298-degree  radial 
and  then  fly  an  approach  into  Keene  Dillant-Hopkins 
(EEN)  airport.  At  the  missed  approach  point,  pilots  were 
instructed  by  air  traffic  control  (ATG)  (the  experimenter) 
to  fly  the  published  missed  approach  course  back  to  the 
Keene  VORTAG  (EEN);  before  reaching  Keene,  they 
were  directed  to  fly  an  approach  into  an  alternate  airport 
(Jaffery,  AFN).  Figure  4  shows  the  area  of  the  flight.  The 
thick  shaded  lines  show  the  basic  flight  segments  used  to 
measure  flight  path  error  during  the  flight. 

The  second  flight  scenario  was  the  same  as  the  first 
flight  except  the  pilots  were  required  to  hold  at  the  LA¬ 
PEL  intersection  on  the  246-degree  radial  instead  of  at 
the  HURLY  intersection.  ATG  then  cleared  the  pilots  for 
an  approach  into  Orange  Gounty  (ORE).  At  ORE,  the 
pilots  flew  the  published  missed  approach.  Pilots  were 
then  instructed  to  fly  an  approach  into  the  alternate 
airport,  KEEN.  Figure  5  depicts  the  flight  area  with 
shaded  lines  indicating  flight  segments  used  to  measure 
flight  path  error. 

During  both  flights,  a  vacuum  system  failure  occurred 
when  the  pilots  were  10  miles  from  the  alternate  airport. 
The  pilots  were  required  to  fly  the  approach  into  the  al¬ 
ternate  airport  using  partial-panel  instrumentation  and/or 


the  GX-50  and  MX-20.  Data  collection  was  terminated 
shortly  before  the  aircraft  reached  the  alternate  airport, 
and  pilots  were  not  required  to  land. 

The  weather  was  set  to  %  of  a  mile  visibility  for  both 
flights.  Winds  were  10  knots  from  260  degrees,  which 
was  a  slight  crossing  headwind  for  most  of  the  flight.  Each 
flight  scenario  lasted  approximately  75  minutes,  and  a 
1 5-minute  break  was  given  to  each  pilot  between  the  flight 
scenarios.  The  order  of  the  GPS  and  VOR  navigation 
conditions  and  flight  plans  was  counter-balanced  across 
participants.  At  the  conclusion  of  the  second  flight,  the 
usability  questionnaire  was  administered  to  the  pilots, 
and  then  they  were  debriefed. 

Hypotheses 

•  It  was  expected  that  measures  of  workload  would 
generally  favor  the  standard  flight  instruments.  This 
expectation  was  based  on  the  literature  (cited  above) 
documenting  the  complexity  of  the  GPS  and  specific 
user-interface  design  problems.  However,  it  was  pos¬ 
sible  that  workload  would  be  mitigated  somewhat  due 
to  the  presence  of  the  MX-20  moving-map  display. 

•  In  contrast  to  workload,  it  was  expected  that  flight 
performance  measures  would  generally  favor  the  GX- 
50/MX-20.  Prior  research  has  shown  that  the  use  of 
horizontal  situation  displays  such  as  the  MX-20  im¬ 
proves  navigation  performance  (Haskell  &  Wickens, 
1993;  Wickens  &  Prevett,  1995). 


Figure  4:  First  flight  scenario. 
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Figure  5:  Second  flight  scenario. 


•  Finally,  it  was  expected  that  the  use  of  the  GX-50/ 
MX-20  displays  would  have  beneficial  effects  on  both 
navigational  performance  and  workload  during  the 
vacuum  pump  failure,  because  the  new  displays  would 
compensate  for  the  resulting  loss  of  information  (e.g., 
loss  of  attitude  indicator  and  heading  indicator). 

RESULTS 

Pilot  Demographics 

Flight  hour  estimates  were  collected  for  a  number  of 
flight  categories.  The  mean,  median,  and  range  of  each 
category  are  presented  in  Table  1 . 

In  addition  to  flight  hours,  participants  were  asked 
to  report  on  their  use  of  personal  computer-based  flight 
simulator  programs  similar  to  the  Microsoft  Flight  Simu¬ 
lator  2002  program  used  in  the  current  study.  Twelve 
of  the  sixteen  pilots  (75%)  had  used  Microsoft  Flight 
Simulator  or  a  similar  program.  Mean  total  use  of  the 
programs  was  estimated  as  21.43  hours.  Median  use  was 
10  hours,  ranging  from  0  to  150  hours. 

Participants  were  also  asked  whether  they  had  used 
any  type  of  GPS  navigation  display  while  flying.  Eleven 
of  the  pilots  (69%)  stated  they  had  used  a  GPS  display. 
Mean  total  use  time  for  those  who  had  used  a  display 
was  25.5  hours.  Median  use  was  16  hours.  Finally,  par¬ 
ticipants  were  asked  whether  they  had  used  a  moving 
map  system  similar  to  the  MX-20  display  used  in  the 


current  study.  Three  of  the  pilots  (19%)  said  they  had 
used  such  a  display  prior  to  the  study. 

Display  Usability 

All  participants  were  asked  to  rate  how  easy  it  was  to 
learn  to  operate  the  display  system.  Ratings  were  obtained 
on  a  5-point  scale  from  1  (very  difficult  to  learn)  to  5  (very 
easy  to  learn).  The  mean  rating  was  3.67,  with  no  rating 
below  a  3  (i.e.,  average  difficulty  to  learn).  Finally,  pilots 
were  asked  whether  they  would  like  to  use  the  navigation 
system  in  a  real  aircraft.  Twelve  of  the  pilots  (75%)  said 
they  would  like  to  use  the  system  in  real  flight. 

After  completing  both  of  the  flights,  participants  filled 
out  the  usability  questionnaire.  MX-20  and  GX-50  fea¬ 
tures  were  rated  according  to  each  of  6  criteria  along  a 
5-point  scale.  A  higher  number  indicated  better  usability 
for  each  of  the  criteria.  The  reader  is  directed  to  the  Ap¬ 
pendix  for  the  definitions  of  the  rating  scale  values.  Table 
2  presents  the  mean  ratings  for  the  MX-20  features  on 
each  of  the  6  usability  criteria. 

In  the  Table,  the  shading  indicates  the  two  highest  rat¬ 
ings  for  a  particular  criterion.  Ties  occurred  for  some  of 
the  criteria.  Only  two  mean  scores  fell  below  3.0.  These 
scores  are  highlighted  using  bold  text  and  borders.  The 
Table  indicates  that  pilots  believed  that  the  map  range 
(zoom)  feature  and  the  map  information  were  the  most 
usable  aspects  of  the  MX-20.  During  normal  operation, 
two  buttons  were  continuously  dedicated  for  zooming. 
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Table  1:  Pilot  Flight  Time  Demographics. 


Demographic  Category 

Mean  Hrs. 

Median  Hrs. 

Range  Hrs. 

Total  Time 

1954.38 

405.00 

145  -  20,000 

Time:  Last  12  Months 

248.75 

170.00 

20  -  950 

Time:  Last  90  Days 

85.73 

80.00 

2-450 

Total  Time:  VFR 

1729.13 

365.00 

115-18,000 

VFR  Time:  Last  12  Months 

222.44 

152.00 

20  -  800 

VFR  Time:  Last  90  Days 

63.50 

38.50 

0-375 

Total  Time:  Actual  Instrument  Flight 

198.89 

6.00 

1  -  2,000 

Actual  Instrument  Time:  Last  12  Months 

13.06 

4.00 

0-85 

Actual  Instrument  Time:  Last  90  Days 

4.34 

1.25 

0-25 

Total  Time:  Simulated  Instruments 

119.06 

52.50 

20-1,000 

Simulated  Instrument  Time:  Last  12  Months 

34.00 

23.50 

0-100 

Simulated  Instrument  Time:  Last  90  Days 

13.69 

4.00 

0-50 

making  it  easy  for  the  pilots  to  alter  the  zoom  level.  The 
lowest  scores  for  all  of  the  MX-20  features  were  those 
related  to  head-down  time. 

In  addition  to  the  functions  listed  in  Table  2,  pilots  were 
asked  specific  questions  about  the  usability  of  a  number 
of  other  features  of  the  MX-20.  Ratings  ranged  from  1 
to  5  on  a  disagree-agree  scale.  These  questions,  and  their 
mean  response  values,  are  listed  in  Table  3. 

Looking  at  Table  3,  we  find  that  most  of  the  items 
were  rated  highly  by  the  pilots.  Items  4,  5,  and  16,  which 
all  dealt  with  situation  awareness,  and  specifically  with 
position  awareness,  were  the  most  highly  rated  items. 
The  lowest-rated  item  was  the  statement,  “Prompts  for 
pilot  inputs,  alerts,  and  advisories  effectively  attracted 
my  attention.”  It  should  be  noted  that  pilot  alerts  and 
advisories  were  visual  only,  with  no  aural  cue  accompany¬ 
ing  the  alert.  Of  the  alerts  and  advisories  available  on  the 
MX-20,  the  only  ones  that  the  pilots  regularly  experienced 
during  the  experiment  were  a  terrain  warning  flag  and  a 
barometric  pressure  confirmation. 

Pilots  also  rated  GX-50  features  according  to  the  same 
six  usability  criteria.  Table  4  presents  the  mean  feature 
ratings  for  the  GX-50. 

The  GX-50  feature  rated  most  usable  was  the  Nearest 
Airport  function.  Pilots  also  were  particularly  satisfied 
with  the  display  of  cross-track  error  data.  As  with  the 
MX-20,  the  lowest  ratings  occurred  for  head-down  time. 
However,  workload  ratings  were  a  close  second. 

The  usability  of  additional  features  was  addressed 
with  specific  questions,  including  four  items  (28-31)  on 
the  effectiveness  of  the  MFD/GPS  displays  during  the 
vacuum  failure  (see  Table  5).  As  before,  these  questions 
were  rated  on  a  disagree-agree  scale  of  1  to  5. 


The  GPS  usability  item  receiving  the  lowest  rating 
concerned  the  ability  of  the  unit  to  attract  pilot  atten¬ 
tion  for  warnings  and  alerts.  The  alerts  received  by  pilots 
during  the  study  were  messages  regarding  the  completion 
of  flight  segments  and  warnings  regarding  approach  to 
restricted  flight  areas.  As  with  the  MX-20,  no  aural  alerts 
were  issued  by  the  GPS. 

In  addition  to  the  specific  usability  questions,  par¬ 
ticipants  were  asked  if  they  had  made  any  errors  while 
interacting  with  the  MX-20  and  GX-50  displays.  Five 
(31%)  of  the  participants  stated  they  had  made  errors 
with  the  MX-20.  Seven  (44%)  participants  admitted 
making  errors  with  the  GX-50.  In  general,  when  asked 
to  comment  on  the  errors  they  committed,  participants 
stated  they  could  not  remember  how  to  perform  particular 
functions  and  so  performed  them  incorrectly.  Many  of 
these  pilots  added  that  they  would  not  have  made  errors 
if  they  had  been  given  more  time  to  learn  how  to  use 
the  displays. 

To  measure  the  pilot’s  subjective  estimates  of  how 
easy  it  was  to  handle  the  vacuum  failure  both  with  and 
without  the  MFD/GPS  equipment,  paired  t-tests  were 
conducted  to  compare  item  28  with  29  and  item  30 
with  31.  Items  28  and  29  asked  the  pilots  to  judge  how 
easy  it  was  to  control  the  aircraft  after  the  vacuum  failure 
while  referencing  only  conventional  navigation  instru¬ 
ments  or  using  the  MFD/GPS.  Items  30  and  31  asked 
the  pilots  to  judge  ease  of  navigating  after  the  vacuum 
failure  using  either  conventional  navigation  instru¬ 
ments  or  the  MFD/GPS.  Both  paired  t-tests  indicated 
significant  differences  between  the  items,  showing  that 
pilots  believed  it  was  easier  to  control  the  aircraft  after 
the  vacuum  failure  using  the  MFD/GPS,  t(13)  =  2.88,  p 


7 


Table  2:  MX-20  mean  feature  ratings  on  six  usability  criteria. 


Eeature  Rating  Criteria 

Part  1.  Multifunction  Display 

Effect- 

Ease  of 

Work- 

Cues  & 

Feed- 

Head 

Row 

iveness 

Use 

load 

Prompts 

back 

Down 

Time 

Mean 

Accessing  database  information 
(runway  length,  frequencies,  etc) 

4.14 

3.50 

3.36 

3.43 

3.86 

2.79 

3.51 

Selecting  tenain  mode 

3.77 

3.46 

3.38 

3.62 

3.69 

2.92 

3.47 

Map  type  selection  (VFR,  IFR, 
Custom) 

4.14 

4.00 

4.00 

3.79 

4.14 

3.64 

3.95 

Adjusting  altimeter  setting 

3.82 

4.36 

4.00 

3.82 

4.00 

3.00 

3.83 

Map  orientation  selection  (North 
up.  Track  up) 

4.23 

3.77 

3.54 

3.85 

3.85 

3.31 

3.76 

Selection  of  map  data  for  display 
-  decluttering 

4.00 

3.86 

3.79 

4.00 

3.86 

3.14 

3.78 

Using  navigation  data 

4.29 

3.93 

3.71 

3.79 

3.64 

3.50 

3.81 

Using  the  map  range  (zoom) 
feature 

4.50 

4.36 

4.21 

4.14 

4.00 

3.86 

4.18 

Using  an  approach  procedure 

4.00 

3.86 

3.36 

3.50 

3.79 

3.14 

3.61 

Selecting  the  Pan  mode 

3.82 

3.82 

3.91 

3.82 

3.55 

3.18 

3.68 

Panning  the  map 

4.09 

4.09 

3.91 

3.73 

3.82 

3.27 

3.82 

Using  map  information 

4.36 

4.29 

4.14 

3.86 

4.00 

3.43 

4.01 

Using  teuain  information 

4.07 

3.71 

3.64 

3.57 

3.50 

3.29 

3.63 

Column  Mean 

3.80 

3.64 

3.50 

3.49 

3.55 

3.03 

<.05,  and  that  it  was  easier  to  navigate  after  the  vacuum 
failure  using  the  MFD/ GPS,  t(13)=4.01,p<.05,  relative 
to  using  conventional  navigation  instruments,  ft  should 
be  noted  that  for  both  tests,  two  of  the  participants  did 
not  respond  to  the  items,  and  so  their  data  were  excluded 
from  the  analysis. 

Display  Interaction  Analysis 

Participants  were  videotaped  during  their  interaction 
with  the  MX-20/GX-50  displays.  These  videotapes  were 
reviewed  so  that  counts  could  be  made  of  the  number  of  ac¬ 
tions  taken  by  participants  to  complete  each  of  their  tasks, 
whether  or  not  they  received  help  from  the  experimenter. 
In  addition,  the  experimenter  noted  whether  assistance 
was  required,  in  the  form  of  explicit  instructions  from 
the  experimenter,  to  complete  a  task.  Excess  actions  were 
computed  by  counting  the  number  of  actions  taken  to 
complete  a  particular  task  and  subtracting  the  minimum 
number  of  actions  required.  Figure  6  shows  the  mean 
number  of  excess  actions  taken  by  participants  by  task. 

Looking  at  Figure  6,  we  see  that  participants  used 
more  excess  button  presses  while  interacting  with  the  GPS 
display  than  when  they  were  interacting  with  the  MFD. 
The  GPS  task  showing  the  most  excess  activations  was 
inserting  an  assigned  intersection  into  the  flight  plan  prior 
to  conducting  a  holding  procedure.  This  task  required 
the  input  of  letters  and  numbers  into  the  display  through 
the  use  of  both  buttons  and  knobs.  Likewise,  the  task 


with  the  second-most  activations,  entering  the  alternate 
airport  into  the  flight  plan,  also  required  the  participants 
to  input  letters  and  numbers  into  the  display. 

In  addition  to  counting  the  number  of  excess  button 
presses,  experimenters  recorded  whether  or  not  pilots 
required  assistance  to  complete  specific  tasks.  Usually, 
such  assistance  consisted  of  explicit  instructions  regarding 
which  buttons  to  press  and  which  knobs  to  turn.  Figure 
7  shows  the  percentage  of  pilots  requiring  assistance  to 
complete  each  of  the  tasks. 

As  with  the  excess  control  activations,  participants  had 
less  trouble  with  the  MFD  than  they  did  with  the  GPS. 
Again,  we  found  that  two  of  the  most  difficult  tasks  for 
the  pilots  involved  entering  letters  and  numbers  into  the 
GPS  display.  However,  for  every  GPS  task,  at  least  50% 
of  the  pilots  required  assistance  to  complete  the  task. 

Subjective  Workload  Estimates 

The  six  individual  workload  estimates  for  a  task  were 
averaged  into  a  single  composite  workload  estimate  for 
that  task.  Figure  8  presents  a  bar  graph  of  the  composite 
scores  for  each  of  seven  tasks,  separated  by  display  type 
(GPS/MFD  vs.  VOR). 

A  two-way  analysis  of  variance  (Flight  Task  x  Display 
Type)  was  conducted  on  the  composite  workload  scores.  The 
only  factor  that  reached  significance  was  Flight  Task,  F(6, 
78)  =  26.94,  p  <.01.  No  effect  for  Display  Type  was  found, 
and  there  was  no  interaction  effect.  A  post  hoc  comparison 


Table  3:  Mean  scores  for  additional  MX-20  (MFD)  usability  items. 


1 .  Overall,  the  GPS/MFD  increased  my  awareness  of  nearby  terrain  relative  to  conventional 
instruments. 

4.40 

2.  The  terrain  proximity  feature  attracted  my  attention  to  terrain  in  a  timely  fashion. 

4.13 

3.  The  GPS/MFD  increased  my  awareness  of  my  ground  speed,  course,  and  altitude  relative 
to  conventional  instruments. 

4.13 

4.  The  GPS/MFD  increased  my  awareness  of  my  position  relative  to  airports,  fixes,  and 
airways  relative  to  conventional  instruments. 

4.87 

5.  The  GPS/MFD  increased  my  awareness  of  my  position  during  instrument  approaches 
relative  to  conventional  instruments. 

4.93 

6.  The  MFD  information  I  needed  was  easy  to  see  and  not  obscured  by  other  information. 

4.40 

7.  Text  displayed  on  the  MFD  was  easy  to  read. 

4.13 

8.  The  meaning  of  symbols  and  text  on  the  MFD  was  easy  to  understand. 

4.40 

9.  Prompts  for  pilot  inputs,  alerts,  and  advisories  effectively  attracted  my  attention. 

3.27 

10.  Prompts  for  pilot  inputs,  alerts,  and  advisories  were  easy  to  understand. 

3.93 

11.  The  MFD  menus  were  easy  to  use. 

4.07 

12.  The  MFD  menu  options  were  easy  to  understand. 

4.07 

13.  The  MFD  button  labels  were  easy  to  understand. 

4.20 

14.  Information  about  fixes  shown  on  the  MFD  maps  was  easy  to  obtain. 

3.80 

15.  The  use  of  colors  on  the  MFD  maps  was  meaningful. 

4.53 

16.  I  remained  geographically  oriented  while  I  changed  heading. 

4.73 

17.  I  remained  aware  of  ownship  position  when  I  panned  the  MFD  map. 

4.21 

18.  The  size  of  the  MFD  display  area  was  adequate  for  the  information  presented. 

4.53 

Mean  Excess  Actions  by  Task 
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Figure  6:  Mean  number  of  excess  actions,  by  task,  taken  by  pilots  using  the  MX-20  and 
GX-50  displays.  Key  -  ifrmap  =  change  to  IFR  map  display;  airwy  =  remove  high  altitude 
airways  from  IFR  map;  nrthtk  =  change  map  display  mode  from  north-up  to  track-up; 
oviytr  =  select  custom  map  and  overlay  relative  terrain;  rtchng  =  set  GPS  to  intercept 
assigned  radial;  inshid  =  insert  assigned  intersection  into  flight  plan  for  holding 
procedure;  sethid  =  select  assigned  outbound  radial  for  holding  procedure;  hidin  =  set 
inbound  radial  for  hold  procedure;  actrted  =  reactivate  auto-sequencing;  Idappi  =  load 
the  first  approach;  misapp  =  activate  the  missed  approach  procedure;  entalt  =  enter  the 
alternate  airport  into  the  flight  plan;  Idapp2  =  load  the  approach  to  the  alternate  airport. 
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Table  4:  GX-50  mean  feature  ratings  on  six  usability  criteria. 


Eeature  Rating  Criteria 

Part  2.  GPS  Navigator 

Effect- 

Ease  of 

Work- 

Cues  & 

Eeed- 

Head 

Row 

iveness 

Use 

load 

Prompts 

back 

Down 

Time 

Mean 

Accessing  database  information 
(runway  lengths,  frequencies, 
etc) 

3.69 

3.38 

3.38 

3.85 

4.00 

2.77 

3.51 

Entering  flight  plans 

4.00 

3.23 

3.15 

3.77 

3.92 

3.08 

3.53 

Editing  flight  plans 

3.69 

3.15 

2.85 

3.54 

3.92 

2.92 

3.35 

Selecting  “Direct  To”  function 

3.92 

3.69 

3.23 

4.15 

4.00 

3.31 

3.72 

Using  GPS  navigator  to  fly  a 
radial  to/from  a  waypoint 

3.69 

3.54 

3.38 

3.62 

3.62 

3.08 

3.49 

Using  GPS  navigator  to  fly  a 
holding  pattern 

3.31 

3.46 

3.00 

3.69 

3.54 

2.92 

3.32 

Loading  and  flying  GPS 
approaches 

3.77 

3.77 

3.38 

3.92 

3.69 

3.08 

3.60 

Using  the  GDI  data 

3.75 

3.58 

3.33 

3.58 

3.83 

3.33 

3.57 

Using  cross-track  error  (XTE) 
data 

3.92 

4.00 

3.83 

3.58 

3.50 

3.58 

3.74 

Using  DTK  data 

3.42 

3.42 

3.25 

3.83 

3.50 

3.33 

3.46 

Using  Message  information 

3.23 

3.23 

3.38 

3.54 

3.69 

3.58 

3.44 

Accessing  flight  plan  leg 
information 

3.85 

3.38 

3.15 

3.92 

4.00 

3.31 

3.60 

Using  the  Nearest  (airport) 
function 

4.25 

3.92 

3.67 

4.08 

4.25 

3.83 

4.00 

Column  Mean 

3.46 

3.27 

3.07 

3.51 

3.53 

3.01 

Table  5:  Mean  scores  for  additional  GX-50  (GPS)  usability  items  and  vacuum  failure  questions. 


1 .  It  was  easy  to  enter  data  into  the  GPS  receiver. 

3.86 

2.  The  knohs  and  buttons  on  the  GPS  receiver  were  placed  in  logical  and  convenient  locations. 

4.21 

3.  The  GPS  receiver  controls  for  different  functions  operate  in  a  consistent  manner. 

4.00 

4.  Text  on  the  GPS  display  was  easy  to  read. 

4.57 

5.  The  size  of  the  GPS  display  area  was  adequate  for  the  information  displayed. 

4.07 

6.  It  was  easy  to  select  GPS  information  (airports,  airport  information,  VORs,  waypoints,  etc.). 

4.00 

7.  It  was  easy  to  select  a  GPS  approach. 

4.29 

8.  GPS  warnings  and  alerts  were  issued  at  the  right  time. 

4.00 

9.  GPS  warnings  and  alerts  effectively  attracted  my  attention. 

3.14 

10.  It  was  easy  to  control  the  airplane  after  the  vacuum  failure  using  the  MFD/GPS. 

4.14 

1 1 .  It  was  easy  to  control  the  airplane  after  the  vacuum  failure  using  only  conventional 
instruments. 

3.14 

12.  It  was  easy  to  navigate  after  the  vacuum  failure  using  the  MFD/GPS. 

4.64 

13.  It  was  easy  to  navigate  after  the  vacuum  failure  using  only  conventional  instruments. 

3.14 
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Figure  7:  Percentage  of  pilots,  by  task,  requiring  assistance  to  complete  specific  user- 
interface  tasks.  Key  -  ifrtnap  =  change  to  IFR  map  display;  airwy  =  remove  high  altitude 
airways  from  IFR  map;  nrthtk  =  change  map  display  mode  from  north-up  to  track-up; 
oviytr  =  select  custom  map  and  overlay  relative  terrain;  rtchng  =  set  GPS  to  intercept 
assigned  radial;  inshid  =  insert  assigned  intersection  into  flight  plan  for  holding  procedure; 
sethid  =  select  assigned  outbound  radial  for  holding  procedure;  hidin  =  set  inbound  radial 
for  hold  procedure;  actrted  =  reactivate  auto-sequencing;  Idappi  =  load  the  first  approach; 
misapp  =  activate  the  missed  approach  procedure;  entalt  =  enter  the  alternate  airport  into 
the  flight  plan;  Idapp2  =  load  the  approach  to  the  alternate  airport. 


of  levels  within  Flight  Task  showed  that  the  last  four  tasks 
were  judged  to  have  produced  significantly  higher  workload 
than  the  first  three  tasks,  t(15)  =  10.048,  p  <  .01. 

Pilot  Performance  Analysis 

A  number  of  flight-performance  variables  were  col¬ 
lected  during  the  flights.  The  most  important  variables 
were  cross-track  error  and  altitude  since  these  are  most 
directly  related  to  distance  from  a  desired  course.  Addi¬ 
tional  performance  data  included  bank,  ground  speed,  and 
heading.  Each  flight  was  divided  into  9  flight  segments 
for  the  purpose  of  analyzing  speciflc  flight-performance 
variables.  Not  all  of  the  segments  were  useful  for  analysis 
and  not  all  of  the  performance  variables  were  valid  for 
every  segment.  For  example.  Segment  1  was  from  takeoff 
until  the  pilot  turned  to  intercept  the  100  degree  radial. 
No  differences  in  course  deviation  were  expected  as  a 
result  of  display  differences  during  this  segment.  Also, 
altitude  changes  continuously  during  an  approach  seg¬ 
ment.  Because  glide-slope  information  was  unavailable, 
the  analysis  of  altitude  during  this  segment  would  not 
be  useful. 


Figure  9  shows  the  root-mean-square  cross- track  error, 
by  display  type,  for  three  flight  segments.  The  inbound 
radial  segment  refers  to  tracking  the  radial  inbound  to¬ 
ward  a  VOR.  After  reaching  the  VOR,  pilots  turned  to  a 
new  heading  and  followed  an  outbound  radial  until  they 
reached  a  designated  intersection.  The  flnal  approach 
segment  occurred  after  the  vacuum  failure  while  the  pilot 
was  flying  with  partial-panel  instrumentation.  Paired  t- 
tests  were  used  to  compare  cross-track  error  differences 
between  display  types  for  each  segment.  The  inbound 
radial  segment  showed  a  signiflcant  difference  according 
to  display  type,  t(15)  =  2.20,  p  <.05,  as  did  the  flnal  ap¬ 
proach  segment,  t(15)  =  3.14,  p  <.05.  In  both  cases,  use 
of  the  GPS/MFD  led  to  signiflcantly  less  error  relative 
to  use  of  the  VOR. 

Much  of  the  difference  in  cross-track  error  can  be 
attributed  to  the  fact  that  the  VOR  signal  spreads  out  as 
it  moves  away  from  the  emitter,  unlike  the  GPS  cross¬ 
track  indication,  which  remains  constant  at  all  positions. 
This  could  explain  why  VOR  flight  performance  on  the 
outbound  radial  showed  much  less  cross-track  error  than 
on  the  inbound  radial:  because  the  outbound  radial  was 
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Figure  8:  Composite  NASA  TLX  subjective  workload  estimates,  by  flight  task  and  display  type. 
PF  -  Pre-Flight;  SIF  -  Set  Initial  Frequencies;  RC  -  Route  Change;  Field  -  Fielding  Procedure; 
MA  -  Missed  Approach;  PTA  -  Proceed  to  Alternate;  VF  -  Vacuum  Failure. 
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Figure  9:  Cross-T rack  error  (in  statute  miles)  by  flight  segment  and  display  type. 
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Figure  10:  Altitude  error  (in  feet)  by  flight  segment  and  display  type. 
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shorter  than  the  inbound  radial  and  would  have  less 
average  error  in  the  signal.  However,  this  explanation 
does  not  account  for  the  large  differences  in  cross-track 
error  found  for  the  final  approach  segment,  where  fac¬ 
tors  related  to  other  differences  in  the  displays  were  most 
likely  involved. 

Figure  10  shows  the  root-mean-square  altitude  error 
for  three  flight  segments.  The  inbound  and  outbound 
radial  segments  are  the  same  as  those  depicted  in  Figure 
9.  The  Hold  segment  occurred  while  the  pilot  was  flying 
the  holding  procedure. 

For  all  three  segments,  altitude  error  was  less  while 
using  the  VOR.  However,  there  was  a  significant  differ¬ 
ence  between  display  types  only  for  the  inbound  radial 
segment,  t(15)  =  2.22,  p  <.05. 

DISCUSSION 

As  expected,  the  pilot  group  recruited  for  the  study 
varied  widely  in  terms  of  age  (21  -  69  years),  total  flight 
experience  (145  -  20,000  hours),  and  most  of  the  other 
experience  variables  that  were  measured  (see  Table  1).  In 
addition,  although  1 1  (69%)  of  the  pilots  had  used  a  GPS 
unit  of  some  type,  only  3  (19%)  had  ever  been  exposed 
to  a  multi-function  display  similar  to  the  MX-20.  This 
variability  in  experience  more  closely  matches  the  gen¬ 
eral  aviation  pilot  population  than  the  professional  pilot 
population  that  has  been  equipped  with  these  displays  as  a 
part  of  the  Capstone  Program  (Williams  et  ah,  2002). 

Results  from  the  usability  questionnaire  demonstrated 
that  pilots  found  both  the  GX-50  and  MX-20  units  to 
be  usable,  without  any  serious  usability  problems.  The 
MX-20  received  higher  usability  scores  on  average  than 
the  GX-50,  but  this  was  expected  because  of  the  simpler 
design  of  the  MX-20  display.  Individual  questions  about 
the  effects  of  the  MX-20  on  situation  awareness  yielded 
strong  positive  responses  from  the  pilots.  Indeed,  the 
two  highest  scores  dealt  with  position  awareness  using 
the  GPS/MFD  relative  to  standard  instruments.  This  is 
a  particularly  important  finding,  given  the  documented 
relationship  between  pilot  situation  awareness  and  flight 
safety  (Endsley,  1995;  Endsley,  1997;  Roscoe,  1968). 

Of  the  six  usability  categories  listed  in  the  questionnaire, 
the  category  receiving  the  lowest  scores  for  both  displays 
was  the  effect  of  the  displays  on  head-down  time.  It  is 
likely  that  increased  practice  and  exposure  to  the  displays 
will  decrease  the  amount  of  head-down  time  required  to 
operate  the  displays.  However,  observations  and  interviews 
of  pilots  using  the  system  have  shown  that  head-down 
time  can  still  be  a  problem  for  specific  functions,  such 
as  the  display  of  traffic  information,  even  after  years  of 
system  use  (Williams  et  al.,  2002).  Some  of  the  lower 


scores  from  the  individual  items  of  both  the  MX-20  and 
GX-50  usability  questions  concerned  the  ability  of  warn¬ 
ings  and  alerts  to  attract  attention.  This  may  be  because 
neither  of  the  displays  presents  auditory  annunciations 
to  the  pilot,  only  visual  annunciations. 

The  analysis  of  pilot  interactions  with  the  displays 
demonstrated  that  the  2  to  3  hours  of  training  received 
by  the  pilots  was  not  enough  to  conduct  the  required 
tasks  without  committing  several  errors  and  requiring 
assistance  from  the  experimenters.  Notably,  the  pilots 
produced  considerably  fewer  errors  and  were  given  much 
less  assistance  when  performing  MX-20  tasks  than  when 
performing  GX-50  tasks.  This  was  expected  because  of 
the  differences  in  the  complexity  of  the  tasks  performed, 
although  it  is  possible  that  an  improved  interface  design 
might  make  the  performance  of  certain  GX-50  tasks  easier. 
These  results  have  implications  regarding  mandatory  FAA 
training  requirements  for  these  displays.  Perhaps  the  FAA 
should  consider  following  the  lead  of  other  countries  in 
this  matter.  Australia,  for  instance,  requires  that  pilots 
attend  an  approved  training  program,  be  flight  tested,  and 
receive  a  license  endorsement  before  flying  a  GPS  display 
in  IFR  conditions  (St.  George  &  Nendick,  1998). 

In  comparing  the  effectiveness  of  the  GPS/MFD  dis¬ 
plays  to  the  standard  VOR  for  conducting  instrument 
flight  tasks,  several  pieces  of  information  are  available 
from  the  study.  First,  there  were  no  significant  differences 
between  the  two  display  types  in  terms  of  the  subjective 
workload  estimates  given  by  the  pilots.  This  was  somewhat 
surprising,  given  the  apparent  difficulty  experienced  by 
many  of  the  pilots  in  performing  the  assigned  tasks  with 
the  GPS/MFD.  One  possible  explanation  is  that  the  pilots 
felt  that  they  could  easily  master  the  GPS/MFD  displays 
with  practice,  which  may  have  affected  their  estimates. 

Despite  the  lack  of  a  significant  difference  between 
subjective  workload  estimates,  performance  differences 
were  found  between  the  two  display  types.  For  instance, 
when  pilots  flew  using  the  GPS/MFD,  they  were  more 
accurate  in  terms  of  cross-track  error  (horizontal  distance 
from  the  intended  flight  path).  As  mentioned  before, 
this  was  expected  given  previous  research  with  moving 
map  displays  (Haskell  &  Wickens,  1993;  Roscoe,  1968; 
Wickens  &  Prevett,  1995).  On  the  other  hand,  pilots 
were  less  accurate  in  holding  altitude  while  flying  the 
GPS/MFD,  at  least  during  the  longest  flight  segment. 
A  possible  explanation  for  this  is  that  the  altimeter  was 
located  closer  to  the  VOR  display  than  the  GPS/MFD 
display,  making  it  more  difficult  for  the  pilots  to  include 
the  altimeter  in  their  scan.  Although  the  MX-20  does 
display  altitude  information,  the  pilots  were  accustomed 
to  looking  for  altitude  information  on  the  altimeter. 
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The  presence  of  the  GPS/MFD  after  the  vacuum 
failure  assisted  pilots  in  maintaining  awareness  of  their 
position  and  in  improving  their  horizontal  flight  path 
performance  during  the  approach  to  the  runway.  Pilots 
flying  the  approach  on  partial-panel  instrumentation  with 
only  the  VOR  showed  a  mean  horizontal  flight  path  er¬ 
ror  of  approximately  1  mile.  In  contrast,  when  they  flew 
the  approach  on  partial-panel  instrumentation  with  the 
assistance  of  the  GPS/MFD,  their  flight  path  error  was 
approximately  of  a  mile;  a  dramatic  improvement. 
Subjective  estimates  further  support  this  flnding.  Pilots 
said  they  felt  that  it  was  both  easier  to  control  the  air¬ 
craft  and  to  navigate  after  the  vacuum  failure  using  the 
GPS/MFD  than  when  using  only  VOR.  These  flndings 
also  support  recent  research  demonstrating  that  the  pres¬ 
ence  of  a  flight  instrument  within  the  primary  held  of 
view  and  containing  a  valid  heading  reference  reduced 
loss  of  control  of  the  aircraft  while  flying  partial-panel 
instrumentation  (Beringer  &  Ball,  2001). 

As  these  new  aircraft  displays  become  more  common 
in  general  aviation  cockpits,  it  is  comforting  to  And  evi¬ 
dence  of  their  effectiveness  and  usability  in  instrument 
flight,  the  most  complicated  of  flight  operations.  Issues 
still  remain,  however.  The  need  for  assistance  while  many 
of  the  pilots  were  interacting  with  the  displays  is  evidence 
that  training  requirements  should  be  modifled  to  specifi¬ 
cally  address  new  technology  in  general  aviation  cockpits. 
Also  poorly  understood  is  how  quickly  knowledge  of  a 
system  is  lost  if  the  pilot  does  not  fly  for  a  period  of  time. 
The  need  for  assistance  also  suggests  that  excessive  head- 
down  time  could  remain  a  problem  when  assistance  is 
not  available.  Flowever,  the  ability  of  the  MX-20/GX-50 
to  help  the  pilot  maintain  situation  awareness,  improve 
navigational  accuracy,  and  increase  safety  in  emergency 
situations  offsets  to  some  extent  the  additional  training 
that  may  be  necessary  or  the  workload  associated  with 
operating  the  displays. 
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APPENDIX  A-  USABILITY  QUESTIONNAIRE 

GPS/MFD  Full-Mission  Usability  Study 

1 .  Please  rate  the  usability  of  each  feature  in  single-pilot  IFR  operations  using  the  rating  scales  in  the 
boxes  below.  For  example,  for  the  first  feature,  choose  one  of  the  items  numbered  1-5  under  A,  Easy 
or  difficult  to  use,  and  write  that  number  in  Box  A  to  the  right  of  the  first  feature  description. 


A.  Effectiveness  of  Feature/Function 

D.  System  cues  and  prompts 

1  =  Very  ineffective 

2  =  Moderately  ineffective 

3  =  Borderline  effective/ineffective 

4  =  Moderately  effective 

5  =  Very  effective 

1  =  Very  ineffective 

2  =  Moderately  ineffective 

3  =  Borderline  effective/ineffective 

4  =  Moderately  effective 

5  =  Very  effective 

B.  Easv  or  difficult  to  use 

1  =  Very  difficult 

2  =  Somewhat  difficult 

3  =  Borderline 

4  =  Easy  to  use 

5  =  Very  easy  to  use 

E.  System  feedback  following  mv  actions 

1  =  Very  ineffective 

2  =  Moderately  ineffective 

3  =  Borderline  effective/ineffective 

4  =  Moderately  effective 

5  =  Very  effective 

C.  Workload 

1  =  Excessive  -  Very  high 

2  =  Excessive  -  Moderately  high 

3  =  Satisfactory  -  Medium 

4  =  Satisfactory  -  Moderately  low 

5  =  Satisfactory  -  Very  low 

F.  Head  down  time  (Effect  on  visual  scan) 

1  =  Excessive  -  Very  negative  effect  on  scan 

2  =  Excessive  -  Moderately  negative  effect  on  scan 

3  =  Satisfactory  -  No  effect  on  scan 

4  =  Satisfactory  -  Moderately  positive  effect  on  scan 

5  =  Satisfactory  -  Very  positive  effect  on  scan 

Feature  Rating  Criteria 

Part  1.  Multifunction  Display 

A 

B 

C 

D 

E 

F 

Accessing  database  information  (runway  lengths,  frequencies,  etc) 

Selecting  terrain  mode 

Map  type  selection  (VFR,  IFR,  Custom) 

Adjusting  altimeter  setting 

Map  orientation  selection  (North  up.  Track  up) 

Selection  of  map  data  for  display  -  decluttering 

Using  navigation  data 

Using  the  map  range  (zoom)  feature 

Using  an  approach  procedure 

Selecting  the  Pan  mode 

Panning  the  map 

Using  map  information 

Using  terrain  information 

Please  feel  free  to  write  a  few  words  below  each  of  the  following  items  to  explain  your  rating. 

1.  Overall,  the  GPS/MFD  increased  my  awareness  of  nearby  terrain  relative  to  conventional  instruments. 
1  2  3  4  5 

Strongly  Somewhat  Neither  Agree  Somewhat  Agree  Strongly  Agree 

Disagree  Disagree  Nor  Disagree 


At 


2.  The  terrain  proximity  feature  attracted  my  attention  to  terrain  in  a  timely  fashion. 

1  2  3  4  5 

Strongly  Somewhat  Neither  Agree  Somewhat  Agree  Strongly  Agree 

Disagree  Disagree  Nor  Disagree 

3.  The  GPS/MFD  increased  my  awareness  of  my  ground 
conventional  instruments. 

1  2  3 

Strongly  Somewhat  Neither  Agree 

Disagree  Disagree  Nor  Disagree 

4.  The  GPS/MFD  increased  my  awareness  of  my  position  relative  to  airports,  fixes,  and  airways  relative  to 
conventional  instruments. 

1  2  3  4  5 

Strongly  Somewhat  Neither  Agree  Somewhat  Agree  Strongly  Agree 

Disagree  Disagree  Nor  Disagree 

5.  The  GPS/MFD  increased  my  awareness  of  my  position  during  instrument  approaches  relative  to  conventional 


instruments. 

1 

2 

3 

4 

5 

Strongly 

Somewhat 

Neither  Agree 

Somewhat  Agree 

Strongly  Agree 

Disagree 

Disagree 

Nor  Disagree 

6. 

The  MFD  information  I  needed  was  easy  to  see  and  not  obscured  hy  other  information. 

1 

2 

3 

4 

5 

Strongly 

Somewhat 

Neither  Agree 

Somewhat  Agree 

Strongly  Agree 

Disagree 

Disagree 

Nor  Disagree 

7. 

Text  displayed  on  the  MFD  was  easy  to  read. 

1 

2 

3 

4 

5 

Strongly 

Somewhat 

Neither  Agree 

Somewhat  Agree 

Strongly  Agree 

Disagree 

Disagree 

Nor  Disagree 

8. 

The  meaning  of  symbols  and  text  on  the  MFD  was  easy  to  understand. 

1 

2 

3 

4 

5 

Strongly 

Somewhat 

Neither  Agree 

Somewhat  Agree 

Strongly  Agree 

Disagree 

Disagree 

Nor  Disagree 

9. 

Prompts  for  ] 

pilot  inputs,  alerts. 

and  advisories  effectively  attracted  my  attention. 

1 

2 

3 

4 

5 

Strongly 

Somewhat 

Neither  Agree 

Somewhat  Agree 

Strongly  Agree 

Disagree 

Disagree 

Nor  Disagree 

10. 

Prompts  for  pilot  inputs,  alerts. 

and  advisories  were  easy  to  understand. 

1 

2 

3 

4 

5 

Strongly 

Somewhat 

Neither  Agree 

Somewhat  Agree 

Strongly  Agree 

Disagree 

Disagree 

Nor  Disagree 

speed,  course,  and  altitude  relative  to 

4  5 

Somewhat  Agree  Strongly  Agree 


A2 


11.  The  MFD 

menus  were  easy  to  use. 

1 

2 

3 

4 

5 

Strongly 

Somewhat 

Neither  Agree 

Somewhat  Agree 

Strongly  Agree 

Disagree 

Disagree 

Nor  Disagree 

12.  The  MFD 

menu  options  were  easy  to  understand. 

1 

2 

3 

4 

5 

Strongly 

Somewhat 

Neither  Agree 

Somewhat  Agree 

Strongly  Agree 

Disagree 

Disagree 

Nor  Disagree 

13.  The  MFD 

button  labels  were  easy  to  understand. 

1 

2 

3 

4 

5 

Strongly 

Somewhat 

Neither  Agree 

Somewhat  Agree 

Strongly  Agree 

Disagree 

Disagree 

Nor  Disagree 

14.  Information  about  fixes  shown  on  the  MFD  maps  was 

easy  to  obtain. 

I 

2 

3 

4 

5 

Strongly 

Somewhat 

Neither  Agree 

Somewhat  Agree 

Strongly  Agree 

Disagree 

Disagree 

Nor  Disagree 

15.  The  use  ol 

'  colors  on  the  MFD  maps  was  meaningful. 

1 

2 

3 

4 

5 

Strongly 

Somewhat 

Neither  Agree 

Somewhat  Agree 

Strongly  Agree 

Disagree 

Disagree 

Nor  Disagree 

16.  I  remained 

1  geographically  oriented  while  I  changed  heading. 

1 

2 

3 

4 

5 

Strongly 

Somewhat 

Neither  Agree 

Somewhat  Agree 

Strongly  Agree 

Disagree 

Disagree 

Nor  Disagree 

17.  I  remained 

1  aware  of  ownship  position  when  I  panned  the  MFD  map. 

1 

2 

3 

4 

5 

Strongly 

Somewhat 

Neither  Agree 

Somewhat  Agree 

Strongly  Agree 

Disagree 

Disagree 

Nor  Disagree 

18.  The  size  of  the  MFD  display  area  was  adequate  for  the  information  presented. 

1 

2 

3 

4 

5 

Strongly 

Somewhat 

Neither  Agree 

Somewhat  Agree 

Strongly  Agree 

Disagree 

Disagree 

Nor  Disagree 

19.  Did  you  make  any  errors  in  using  the  MFD  (any  misinterpretations  of  displayed  information,  or  anything  you 
did  incorrectly  or  omitted)?  Yes  No 

If  so,  please  descrihe  the  error(s)  as  well  as  the  cause  and  consequences  of  those  errors: 
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A.  Effectiveness  of  Feature/Function 

D.  System  cues  and  prompts 

1  =  Very  ineffective 

2  =  Moderately  ineffective 

3  =  Borderline  effective/ineffective 

4  =  Moderately  effective 

5  =  Very  effective 

1  =  Very  ineffective 

2  =  Moderately  ineffective 

3  =  Borderline  effective/ineffective 

4  =  Moderately  effective 

5  =  Very  effective 

B.  Easy  or  difficult  to  use 

1  =  Very  difficult 

2  =  Somewhat  difficult 

3  =  Borderline 

4  =  Easy  to  use 

5  =  Very  easy  to  use 

E.  System  feedback  following  my  actions 

1  =  Very  ineffective 

2  =  Moderately  ineffective 

3  =  Borderline  effective/ineffective 

4  =  Moderately  effective 

5  =  Very  effective 

C.  Workload 

1  =  Excessive  -  Very  high 

2  =  Excessive  -  Moderately  high 

3  =  Satisfactory  -  Medium 

4  =  Satisfactory  -  Moderately  low 

5  =  Satisfactory  -  Very  low 

F.  Head  down  time  (Effect  on  visual  scan) 

1  =  Excessive  -  Very  negative  effect  on  scan 

2  =  Excessive  -  Moderately  negative  effect  on  scan 

3  =  Satisfactory  -  No  effect  on  scan 

4  =  Satisfactory  -  Moderately  positive  effect  on 
scan 

5  =  Satisfactory  -  Very  positive  effect  on  scan 

Part  2.  GPS  Navigator 

A 

B 

C 

D 

E 

F 

Accessing  database  information  (runway  lengths,  frequencies,  etc) 

Entering  flight  plans 

Editing  flight  plans 

Selecting  “Direct  To”  function 

Using  GPS  navigator  to  fly  a  radial  to/from  a  waypoint 

Using  GPS  navigator  to  fly  a  holding  pattern 

Loading  and  flying  GPS  approaches 

Using  the  GDI  data 

Using  cross-track  error  (XTE)  data 

Using  DTK  data 

Using  Message  information 

Accessing  flight  plan  leg  information 

Using  the  Nearest  (airport)  function 

Please  feel  free  to  write  a  few  words  below  each  of  the  following  items  to  explain  your  rating. 

20.  It  was  easy  to  enter  data  into  the  GPS  receiver. 

1  2  3  4  5 

Strongly  Somewhat  Neither  Agree  Somewhat  Agree  Strongly  Agree 

Disagree  Disagree  Nor  Disagree 

21.  The  knohs  and  buttons  on  the  GPS  receiver  were  placed  in  logical  and  convenient  locations. 

1  2  3  4  5 

Strongly  Somewhat  Neither  Agree  Somewhat  Agree  Strongly  Agree 

Disagree  Disagree  Nor  Disagree 
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22.  The  GPS  receiver  controls  for  different  functions  operate  in  a  consistent  manner. 


1 

2 

3 

4 

5 

Strongly 

Somewhat 

Neither  Agree 

Somewhat 

Strongly 

Disagree 

Disagree 

Nor  Disagree 

Agree 

Agree 

23.  Text  on  the  GPS  display  was  easy  to  read. 

1 

2 

3 

4 

5 

Strongly 

Somewhat 

Neither  Agree 

Somewhat  Agree 

Strongly  Agree 

Disagree 

Disagree 

Nor  Disagree 

24.  The  size  of  the  GPS  display  area  was  adequate  for  the 

information  displayed. 

1 

2 

3 

4 

5 

Strongly 

Somewhat 

Neither  Agree 

Somewhat  Agree 

Strongly  Agree 

Disagree 

Disagree 

Nor  Disagree 

25.  It  was  easy 

to  select  GPS  information  (airports,  airport  information,  VORs,  waypoints,  etc.). 

1 

2 

3 

4 

5 

Strongly 

Somewhat 

Neither  Agree 

Somewhat  Agree 

Strongly  Agree 

Disagree 

Disagree 

Nor  Disagree 

26.  It  was  easy 

to  select  a  GPS  approach. 

1 

2 

3 

4 

5 

Strongly 

Somewhat 

Neither  Agree 

Somewhat  Agree 

Strongly  Agree 

Disagree 

Disagree 

Nor  Disagree 

27.  GPS  warnings  and  alerts  were  issued  at  the  right  time. 

1 

2 

3 

4 

5 

Strongly 

Somewhat 

Neither  Agree 

Somewhat  Agree 

Strongly  Agree 

Disagree 

Disagree 

Nor  Disagree 

28.  GPS  warnings  and  alerts  effectively  attracted  my  attention. 

1 

2 

3 

4 

5 

Strongly 

Somewhat 

Neither  Agree 

Somewhat  Agree 

Strongly  Agree 

Disagree 

Disagree 

Nor  Disagree 

29.  It  was  easy 

to  control  the  airplane 

after  the  vacuum  failure  using  the  MFD/GPS. 

1 

2 

3 

4 

5 

Strongly 

Somewhat 

Neither  Agree 

Somewhat  Agree 

Strongly  Agree 

Disagree 

Disagree 

Nor  Disagree 

30.  It  was  easy 

to  control  the  airplane 

after  the  vacuum  failure  using  only  conventional  instruments. 

1 

2 

3 

4 

5 

Strongly 

Somewhat 

Neither  Agree 

Somewhat  Agree 

Strongly  Agree 

Disagree 

Disagree 

Nor  Disagree 

3 1 .  It  was  easy 

to  navigate  after  the  vacuum  failure  using  the  MFD/GPS. 

1 

2 

3 

4 

5 

Strongly 

Somewhat 

Neither  Agree 

Somewhat  Agree 

Strongly  Agree 

Disagree 

Disagree 

Nor  Disagree 

32.  It  was  easy 

to  navigate  after  the  vacuum  failure  using  only  conventional  instruments. 

1 

2 

3 

4 

5 

Strongly 

Somewhat 

Neither  Agree 

Somewhat  Agree 

Strongly  Agree 

Disagree 

Disagree 

Nor  Disagree 

33.  Did  you  make  any  errors  in  using  the  GPS  (any  misinterpretations  of  displayed  information,  or 
anything  you  did  incorrectly  or  omitted)?  Yes  No 

If  so,  please  descrihe  the  error(s)  as  well  as  the  cause  and  consequences  of  those  errors: 
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